Estimation of High-dimensional Prior and Posterior Covariance Matrices in Kalman Filter Variants

This work studies the effect of using Monte Carlo based methods to estimate high-dimensional systems. Recent focus in the geosciences has been on representing the atmospheric state using a probability density function, and, for extremely high-dimensional systems, various sample based Kalman filter techniques have been developed to address the problem of real-time assimilation of system information and observations. As the employed sample sizes are typically several orders of magnitude smaller than the system dimension, such sampling techniques inevitably induces considerable variability into the state estimate, primarily through prior and posterior sample covariance matrices. In this article we quantify this variability with mean squared error measures for two Monte-Carlo based Kalman filter variants, the ensemble Kalman filter and the square-root filter. Under weak assumptions, we derive exact expressions of the error measures. In other cases, we rely on matrix expansions and provide approximations. We show that covariance-shrinking (tapering) based on the Schur product of the prior sample covariance matrix and a positive definite function is a simple, computationally feasible, and very effective technique to reduce sample variability and to address rank-deficient sample covariances. We propose practical rules for obtaining optimally tapered sample covariance matrices. The theoretical results are verified and illustrated with extensive simulations.